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Google Cloud 


Organize the world's 
information and make it 
universally accessible 
and useful. 


Google's Mission 



8.4B 

The number of connected “things” 
in use in 2017, up 31% from 2016* 


We're generating more data than ever before 


Google Cloud Platform 
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loT is a period of transformation 
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loT is a period of transformation 


Not 

Connected 



Information is Everywhere But it’s not DATA Yet 



Transportation 


Buildings 


Manufacturing & Industrial 


Healthcare 












How do you collect and process this analog information, to transform 
into useful business Intelligence? 


Information Insight 





How do you collect and process this analog information, to transform 
into useful business Intelligence? 


Information Insight 





Digital 


Analog 




How do you collect and process this analog information, to transform 
into useful business Intelligence? 


Information Insight 





Digital 


Analog 





A datacenter is not a collection of computers|: 
^ a datacenter is a connputer. 
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Laying undersea cable 
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Google Innovations in Networking 
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Google Jupiter : 

On the left is a Jupiter superblock. It is 
one part of a fabric which interconnect up 
to 100,000 servers at 10 Gbit/s 
each—nnore than 1 Petabit/sec of total 
bisection bandwidth for transfer of 
infornnation between physical and virtual 
nnachines It's enough to transfer the entire 
scanned contents of the library of 
congress in 1 /10 of a second. 
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Server 
racks 
with ToR 
switches 


jd Platform 
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Google Network 

More than a collection of data centers 



(JP, HK, SG) 2013 


• Points of presence (>100) 
■ Network fiber 

• Google Global Cache 
(GGC) edge nodes 


Unity (US, JP) 2010 


FASTER (US, JP,TW)2016 













Our research and legacy in data management 
and analytics stack run deep 



IhMSsis;^ 


i APACHE 


DRILL 
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Open Source 


Google Cloud Platform 
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Our research and legacy in data management 
and analytics stack run deep 

GFS BigTable Flume/Java Millwheel TensorFlow 


MapReduce 

0 

Dremel 

P 


BigQuery 


Bigtable 

Spanner 
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PubSub 

Spanner 

Pub/Sub 

Dataflow 

ML 
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2016 


Google Papers + Products 


Google Cloud Platform 


16 



















Google Big Data Pattern 


Analyze 


Use 


Capture 




Google App 
Engine 



Cloud Logs 



Google Analytics 
Premium 




Cloud Pub/Sub 


Cloud 

Monitoring 


Process 


Cloud Dataflow 



Real-time 


Real-time 

dashboard 


alerts 


Store 


— Batch — 
Stream —>► 



BigQuery Storage 
(tables) 



Cloud Bigtable 
(NoSQL) 



Cloud Storage 
(files) 



Data Studio 


+ a b I e a u 

QlikQ 

iCharts 



Analysts 



Data 

Scientists 


[Hi 

V 

Smart 

apps 


Catalog & Data Lifecycle Automation 









































loT Core handles device management and bi-directional device communication as 

part of an over all GCP loT Solution 


Devices 





What Services are in loT-Core ? 


Device Manager 

• Maintains registries of devices as 
named logical resources 

• Protects these entries with lAM 
permissions 

o eg who can delete a device 

• Associates device credentials with these 
resources 

• Acts as the identity provider (IdP) for the 
MQTT broker 

• Provides storage and delivery API for 
device configurations 

• Maintains some operational state 
metadata for the device: 

o enabled/disabled 
o connectivity and error status 


MQTT Broker 

• Provides a stateful socket connection to 
devices for bidirectional communication 

• Brokers device->cloud telemetry data 
onto Cloud PubSub for use in 
downstream GCP products or customer 
applications 

• Delivers configuration updates via a 
Device Manager API 

• Is exposed through a global DNS 
endpoint over multiple ports 


Simple Ul for monitoring and management 


= Google Cloud Platform 


^ Home 
BIG DATA 



BIgQuery 


Pub/Sub 


Dataproc 

ffj 

Se' 

Dataflow 


ML Engine 


Genomics 


loT Core 



ioT Core 

Device registries 

D CREATE DEVICE REGISTRY 

SHOW INFO PANEL 


Registry ID 

Region 

Protocol 

Topic 


group-register 

us-central1 

MQTT 

projects/gcp-io-demo/topics/group-register-events 


group-test 

mcu-registry 


weather-station 


us-central1 

us-central1 

us-central1 


MQTT 

MQTT 

MQTT 


projects/gcp-io-demo/topics/group-test-events 

projects/gcp-to-demo/topics/weather-statior>-evenis 

projects/gcp-io-demo/topics/weather-statiort-events 


loT Core ^ Registry details x edit registry g dliete 

weather-station 

Regio- us-centrall MQTT ^ub/Sub topic projects/gcp-io-demo/topics/weather-siation-events 

View in Stackdriver 


Add device 


Last SMn 

May?. 2017.10:37:21 PM 
May 7. 2017.10:37:26 PM 
May 4. 2017.10:27:00 AM 


Registered devices 

bevies ID 
b827ebe81110 
indranil-at-test 
rpi-aaron 


state 

Ct Enabled 
^ Enabled 
A Disabled 


Google Cloud Platform 
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^What is MQTT 



MQTT is a machine-to-machine (M2M)/"Internet of Things" connectivity protocoi. 
It was designed as an extremeiy iightweight pubiish/subscribe messaging 
transport. 


• Originally developed at IBM in 1999 and designed for constrained devices 

• Is now a ratified standard 

• Is a binary protocol making efficient use of over-the-wire bandwidth 

• Is simple to implement and so has many small memory footprint libraries available 

• Has become a common and de-facto standard used in many loT projects 





Device Manager Resource Model 


The device manager organizes cloud resources to handle device 
management. 






















Device Manager: Devices : Identity 


((*)) ^ 

Device Provisioner Device Manager 



Device identity is based on an asymmetric 
key-pair of two supported formats: 

• RSA 256 public key wrapped in a X.509v3 
certificate 

• Elliptic curve (ECDSA) algorithm using 
P-256 and SHA-256 [more efficient, better 
suited for small devices] 

Credentials may optionally have an expiration 
timestamp 

A device can have up to 3 credentials associated 
with it at a time, allowing for rotation 

The service should never need the private key 


The sequence shown here is only one way to 
handle device provisioning 











MQTT Broker: authentication 



Device 


MQTT Bridge 



A device authenticates with two pieces of information: 

1. the MQTT client ID (a feature of the MQTT protocol) 
which must be in the form of device name: 

projects/{project-id}/locations/{cloud-region}/registries/{registry-id}/devi 

ces/{device-id} 

2. An MQTT password in the form of a JWT token signed 
by the device's private key 

• The "username" field in MQTT clients is ignored 

• JWT token may have a max expiration of 1 hour 

• Device's clock must be within 10 minutes of Google's 
time (use Google NTP) 








Ill 


Dataflow; organize torrents of loT data 
into actionable windows 




^ Google Cloud Platform 














Aggregation 











Unbounded 






Processing Time 10:00 10:02 10:04 10:06 


Sum 
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Where in event time? 


Windowing divides data into event-time-based finite chunks. 

Fixed Sliding Sessions 



Time 

Often required when doing aggregations over unbounded data. 


What Where When How 




















Processing Time 


When in processing time? 


▲ 


Skew 


{-= 


Reality 

Watermark) 


i 




Ideal 


Event Time 


• Triggers control 
when results are 
emitted. 

• Triggers are often 
relative to the 
watermark. 


What Where When How 




Processing Time 

12:06 12:07 12:08 12:09 12:10 


When; Triggering at the Watermark 


Perfect Watermark 



12:00 12:01 12:02 12:03 12 :oa 12:05 12:06 12:07 12:08 12:09 

Event Time 

Perfect watermark: .......... 

Ideal watermark: . 


Heuristic Watermark 

o 

St 



Heuristic watermark: 
Ideal watermark: 


12:09 


What Where When How 






















Customizing What When Where How 



1.Classic Batch 2. Batch with Fixed 

Windows 



12»0 12WI ia;02 12;03 12;iX 12:05 i2;o5 12:07 12:o8 12:09 

Event Time 


Heuristic watermark: 

Ideal watermark; . 

3. Streaming 



12»0 12»1 12:02 12:03 12:04 12:05 12:o5 12»7 12:08 12:09 

Event Time 


Heuristic watermark: 

Ideal watermark: 

4. Streaming with 
Speculative + Late Data 



12:00 12:01 12:02 12:03 12:04 12:05 12:06 12:07 12:08 l 2 :og 

Event Time 


Heuristicwatermark: 

Ideal watermark; . 

5. Streaming With 
Accumulations 


What Where When How 








The BCOrn Model & Cloud Dataflow 


Apache Beam 





Google Cloud Dataflow 

A 

• • • 


a unified model for 
batch and stream processing 

supporting multiple runtimes 


a great place to run Beam 



one important technology we use is neural networks 
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OUTPUT 
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neural net models learn from examples 
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"cat" 


"dog" 


OUTPUT 


"apple" 


"flower" 


labeled photos 
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neural net models learn from examples 


Make tiny adjustments to 
model so output is closer to 
label for a given image 
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"cat" 


"dog" 


OUTPUT 


"apple" 


"flower" 


labeled photos 
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after a model is trained, you can test it 
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unlabeled photo 









after a model is trained, you can test it 
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unlabeled photo 
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Can I Hug That? 




Google Cloud Platform 


38 



powerful functions that neural nets can learn 


Input 

Output 

i 



"rice" 

s. . . -S' 

E £ S E 1. ; U 

"restaurants in Seoul" 

o}'smM\s. 

"hello!" 

) 

1 



"A close up of a small child holding a stuffed animal." 










Rapidly accelerating use of deep learning 
at Google 


Number of directories containing model description files 



2012 2013 2014 2015 


Used across products: 

G A > 

e 0 A 

A a M 

^ G3 


Google Cloud 
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Our new Cloud TPU delivers up to 180 teraflops to train 

and run machine learning models. 







99999 


99999 


666^9 


99666 


66666 


66666 




Get started by visiting - 
cloud.google.conn/loT-Core 


Google Cloud Platform 
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